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Abstract 


This  document  presents  a  preliminary  heuristic  evaluation  of  the  CommandView  user  interface  in 
use  at  the  National  Defence  Command  Centre.  The  analysis  is  based  on  Nielsen’s  Heuristics.  In 
general,  CommandView  displays  key  information  for  the  Joint  staff  in  a  comprehensive, 
integrated  format.  However,  key  problems  involve  consistency  of  organization.  A  solution  may 
be  to  provide  a  consistent  interface  by  providing  standardized  terminology  across  sites  and 
dynamic  control  of  content.  A  proper  evaluation  study  involving  multiple  evaluators  and/or 
usability  testing  is  recommended  to  generate  more  representative  and  reliable  results  and  generate 
further  guidance  for  improvements. 


Resume 


Ce  document  presente  les  resultats  de  Tanalyse  preliminaire  des  interfaces  de  CommandView. 
L’analyse  est  guidee  par  Tapproche  heuristique  de  Nielsen.  Generalement,  CommandView  fourni 
Tinformation  indispensable  aux  membres  de  Tetat  major  inter  armes  d’une  maniere  complete  et 
integree.  Par  contre,  la  coherence  de  Torganisation  reste  un  probleme  majeur.  La  coherence  des 
interfaces  afm  d’offrir  une  terminologie  standards  au  travers  des  sites  et  le  control  dynamique  de 
contenu  pourraient  s’averer  comme  une  solution  potentielle.  Une  evaluation  plus  rigoureuse  est 
requise  en  cherchant  a  impliquer  plusieurs  evaluateurs  par  exemple.  Des  tests  d’utilisation  sont 
recommandes  afm  de  generer  des  resultats  plus  fiables  et  des  recommandations  d’ amelioration 
plus  fondees. 
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Executive  summary 


CommandView  user  interface  evaluation 

Hollands,  J.  G.;  DRDC  Toronto  TM  2006-039;  Defence  R&D  Canada  -  Toronto; 
March  2006. 

A  preliminary  heuristic  interface  evaluation  of  CommandView  (a  system  used  by  the  Joint  staff 
on  the  TITAN  network)  was  conducted  based  on  Nielsen’s  heuristics.  This  involved  considering 
the  user  interface  with  reference  to  a  fixed  set  of  usability  heuristics  derived  from  a  factor 
analysis  of  many  usability  problems.  The  ten  (10)  heuristics  included: 

•  Visibility  of  system  status, 

•  Match  between  system  and  real  world, 

•  User  control  and  freedom, 

•  Consistency  and  standards, 

•  Error  prevention, 

•  Recognition  rather  than  recall, 

•  Flexibility  and  efficiency  of  use, 

•  Aesthetic  and  minimalist  design, 

•  Help  users  recognize,  diagnose  and  recover  from  errors,  and 

•  Help  and  documentation. 

In  general  CommandView  displays  key  information  for  the  Joint  staff  in  a  comprehensive 
integrated  format.  However,  key  problems  involve  consistency  of  organization.  A  solution  may 
be  to  provide  a  consistent  interface  by  providing  standardized  terminology  across  sites  and 
dynamic  control  of  content. 

A  proper  evaluation  study  involving  multiple  evaluators  and/or  usability  testing  is  recommended 
to  generate  more  representative  and  reliable  results  and  generate  further  guidance  for 
improvements. 
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Sommaire 


CommandView  user  interface  evaluation 

Hollands,  J.  G.;  DRDC  Toronto  TM  2006-039;  R  &  D  pour  la  defense  Canada  - 
Toronto;  March  2006. 

Une  evaluation  heuristique  preliminaire  du  CommandView  a  ete  effectuee  selon  les  heuristiques 
de  Nielsen.  L’evaluation  consistait  a  examiner  l’interface  utilisateur  par  rapport  a  une  serie 
d’heuristiques  d’utilisation  derivees  d’une  analyse  factorielle  de  nombreux  problemes 
d’utilisation.  Les  dix  (10)  heuristiques  etaient : 

•  Visibilite  du  statut  du  systeme, 

•  Compatibilite  entre  systeme  et  monde  reel 

•  Controle  par  l’utilisateur  et  liberte  de  l’utilisateur 

•  Coherence  et  normes 

•  Prevention  des  erreurs 

•  Reconnaissance  plutot  que  rappel 

•  Flexibilite  et  efficacite  d’utilisation 

•  Esthetique  et  conception  minimaliste 

•  Aider  les  utilisateurs  a  reconnaitre,  a  diagnostiquer  et  a  reparer  les  erreurs 

•  Aide  et  documentation 

De  maniere  generate,  le  CommandView  affiche  des  informations  importantes  pour  l’etat-major 
interarmees  dans  un  format  integre  complet.  Mais,  les  principaux  problemes  relevaient  de  la 
coherence  de  F organisation.  On  pourrait  peut-etre  y  remedier  a  l’aide  d’une  interface  coherente 
assuree  par  l’utilisation  d’une  terminologie  normalisee  dans  tous  les  sites  et  un  controle 
dynamique  du  contenu. 

Une  etude  d’ evaluation  appropriee  effectuee  avec  des  evaluateurs  multiples  et/ou  des  essais  de 
convivialite  est  a  conseiller  afm  de  produire  des  resultats  plus  significatifs  et  plus  fiables  et 
elaborer  d’autres  directives  pour  apporter  des  ameliorations. 
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1.  Background 


This  document  presents  a  preliminary  heuristic  evaluation  of  the  user  interface  of  a  system  called 
CommandView  used  by  the  Joint  Staff  on  the  TITAN  system.  At  the  encouragement  of  the  Lead 
Scientist,  JCDS  21  TDP  (Dr  Guitouni)  the  process  started  with  an  initial  informal  evaluation, 
which  involved  access  from  a  TITAN  terminal  at  DRDC  Valcartier.  Then  a  more  formal  heuristic 
evaluation  process  was  followed  [1],  This  involves  considering  the  user  interface  with  reference 
to  a  fixed  set  of  usability  heuristics  derived  from  a  factor  analysis  of  many  usability  problems. 
This  approach  generally  provides  useful  guidance  for  designers  towards  producing  an  effective 
user  interface  for  a  wide  variety  of  software. 

For  this  inspection  the  author  acted  as  the  single  evaluator.  The  evaluation  was  conducted  on  June 
29,  2005.  However,  for  a  proper  heuristic  evaluation  multiple  evaluators  should  perform 
evaluations  independently.  Furthermore,  the  best  method  for  evaluating  the  utility  of  an  interface 
is  to  conduct  usability  testing,  which  involves  having  actual  users  attempt  to  perform  a  range  of 
representative  tasks  using  the  interface  while  performance  measures  are  collected.  Given  the 
logistical  difficulties  in  setting  something  like  this  up  within  a  short  time  interval,  it  was  decided 
to  provide  at  least  some  feedback  on  the  user  interface  for  CommandView,  with  the  caveat  that 
the  evaluation  process  was  somewhat  subjective.  A  more  comprehensive  evaluation  study 
involving  usability  testing  or  other  more  formal  methods  should  be  conducted  to  generate  more 
robust  analysis  results  and  generate  guidance  for  improvements. 

It  would  have  been  useful  for  the  evaluator  to  have  had  a  better  understanding  of  the  typical 
user’s  tasks  and  situation.  The  author’s  working  assumption  was  that  CommandView  is  used  to 
see  the  current  picture — to  update  on  the  current  jointsituation,  the  organizations  that  affect  it  and 
the  sources  that  could  provide  information  about  it. 
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2.  General  description  of  CommandView  interface 


CommandView  serves  as  a  kind  of  portal  to  information  relevant  to  the  Joint  Staff  ( J Staff)  and 
Canadian  Forces  Headquarters  (CFHQs).  Figure  show  CommandView  main  pages.  The  main 
page  includes  a  map  depicting  various  incidents  around  the  world,  with  a  more  detailed  National 
View  on  request.  Left  of  the  map  are  links  to  various  elements  of  the  JStaff,  Force  Generation, 
Allied  Agencies,  Standard  Operating  Procedures  (SOPs),  and  Other  Government  Departments 
(OGDs).  On  the  right  are  the  map  legend  and  various  reference  sources,  including  the  Global 
Situation  Brief,  Incident  Management  System,  Read  File,  Knowledge  Base  etc.  Clicking  on  a 
button  brings  up  a  new  page,  most  commonly  from  another  site  (typically  the  NDCC  site).  In  this 
sense  CommandView  functions  as  a  kind  of  centralized  access  or  portal  to  other  sites. 


Comments 
Comd  Notes 


Sytoey 

Ottawa 

00  41  06 

1041  06 

Figure  1.  CommandView:  International  Situation  Viewl 


'Unclassified  View  of  CommandView:  Source  DWAN. 
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Esquimau  Edmonton  Regina 


0949  47  |  10-49-47  |  11  4»47  |  11  4»47  |  124947  |  134947 


t0«»4f  |  124»47 


Figure  2.  CommandView:  Domestic  Situation  View  2 


In  general  the  main  page  succeeds  as  a  method  for  bringing  together  relevant  sources  in  one 
location — “one  stop  shopping”  for  the  JStaff.  The  ticker  tape  text  that  highlights  recent  events  is  a 
nice  touch.  There  are  multiple  access  points  to  information  and  the  map  provides  a  useful,  easily 
understandable  view  on  current  activities  and  events,  and  also  a  method  for  switching  between 
domestic  and  international  views.  Recent  Situation  Report  (Sitrep)  briefings  are  easily  accessed. 


2  Idem. 
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3.  Nielsen’s  usability  heuristics  applied  to 
CommandView 


In  this  section,  we  present  a  description  of  ten  (10)  heuristics  proposed  by  Nielsen  [1]  and 
followed  by  pertinent  comments  applicable  to  CommandView. 

1.  Visibility  of  system  status:  The  system  should  always  keep  users  informed  about  what  is 
going  on,  through  appropriate  feedback  within  reasonable  time. 

CommandView:  As  a  user,  it  is  not  clear  when  one  has  left  the  CommandView  System.  In 
fact,  almost  all  the  content  appears  to  be  accessed  from  other  sites.  This  leads  to  certain 
problems: 

♦  Inconsistency:  As  soon  as  the  user  leaves  the  main  CommandView  page,  its 
organizational  structure  is  gone,  replaced  by  whatever  organization  is  available  from 
the  remote  site.  This  could  lead  to  problems  with  users  getting  lost.  Beyond 
organization,  there  are  appearance  and  terminology  changes  across  sites. 

♦  Delays:  There  are  (probably  unavoidable)  delays  while  connecting  to  other  sites. 
There  is  a  need  to  ensure  that  these  delays  are  minimized,  and  when  this  is  not 
possible,  that  the  user  is  given  an  estimate  of  time  remaining  or  some  indication  of 
system  status  as  the  user  waits  for  a  page  to  load. 

2.  Match  between  system  and  real  world.  The  system  should  speak  the  users’  language,  with 
words,  phrases,  and  concepts  familiar  to  the  user,  rather  than  system-oriented  terms.  Follow 
real-world  conventions,  making  information  appear  in  a  natural  and  logical  order. 

CommandView:  Within  the  CommandView  pages,  the  terms  used  seem  to  be  sensibly 
chosen  and  appropriate  for  the  user  population.  However,  they  are  not  always  consistent 
with  those  found  on  other  pages. 

Questions  of  organization  also  affect  how  well  a  system  speaks  the  user’s  language.  I  found 
the  organization  of  the  CommandView  main  page  unclear  initially.  There  seemed  to  be  little 
rationale  for  why  particular  buttons  were  placed  in  particular  locations.  No  titles  were 
provided  for  some  columns  of  buttons,  and  there  appears  little  rationale  for  the  choices  of 
what  is  positioned  where  (left/right,  up/down)  and  what  gets  grouped  together.  When  I 
found  the  Site  Map  I  found  links  to  some  pages  that  I  did  not  see  from  the  main  page.  Site 
Map  headings  included:  JStaff  Nav;  Force  Gen  Nav;  Site  Nav.  I  think  an  organization  like 
this  might  help  the  main  page — simply  having  the  names  may  help  the  user  group  or 
categorize  the  buttons. 

3.  User  control  and  freedom.  Users  often  choose  system  functions  by  mistake  and  will  need  a 
clearly  marked  “ emergency  exit ”  to  leave  the  unwanted  state  without  having  to  go  through  an 
extended  dialogue.  Support  undo  and  redo. 

CommandView:  The  “Watch  notes”  (otherwise  a  nice  idea)  pop-up  over  the  map,  which 
blocks  the  view  of  the  rest  of  the  map  and  are  not  movable.  One  should  allow  the  user  to 
move  the  watch  note  aside.  A  graphical  representation  of  the  linkages  between  sites  (using 
frames,  for  example)  would  be  useful. 

4.  Consistency  and  standards.  Users  should  not  have  to  wonder  whether  different  words, 
situations,  or  actions  mean  the  same  thing.  Follow  platform  conventions. 
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CommandView:  The  CommandView  pages  consistently  follow  the  Government  of  Canada 
(GOC)  web  site  conventions.  Again  there  are  problems  with  the  consistency  of  information 
presentation  when  visiting  non-GOC  sites. 

There  were  observed  problems  with  the  consistency  of  clock  times.  The  time  zone  clocks 
and  Windows  clock  did  not  correspond,  and  neither  corresponded  to  the  actual  real  time 
during  the  evaluation  (in  one  case  deviating  by  more  than  an  hour).  This  may  have  been  a 
problem  with  the  local  machine.  Users  will  have  difficulty  interpreting  and  relying  on 
displayed  times  if  different  clocks  are  inconsistent. 

5.  Error  prevention.  Even  better  than  good  error  messages  is  a  careful  design  which  prevents  a 
problem  from  occurring  in  the  first  place. 

CommandView:  As  a  web  interface,  there  are  few  error  messages  except  those  that  occur 
when  a  page  cannot  load.  There  are  few  opportunities  for  data  entry,  for  which  lost  data  is  a 
common  concern. 

6.  Recognition  rather  than  recall.  Make  objects,  actions,  and  options  visible.  The  user  should 
not  have  to  remember  information  from  one  part  of  the  dialogue  to  another.  Instructions  for 
use  of  the  system  should  be  visible  or  easily  retrievable  whenever  appropriate. 

CommandView:  A  general  problem  with  web  interfaces  is  that  the  user  sees  a  single  page 
at  a  time.  This  means  that  if  information  on  one  page  is  relevant  to  that  on  another  it  will 
need  to  be  remembered  by  the  user.  Although  it  is  possible  to  open  multiple  web  pages  at 
the  same  time,  managing  all  opened  pages  may  become  problematic. 

7.  Flexibility  and  efficiency  of  use.  Accelerators — unseen  by  the  novice  user — may  often  speed 
up  the  interaction  for  the  expert  user  to  such  an  extent  that  the  system  can  cater  to  both 
inexperienced  and  experienced  users.  Allow  users  to  tailor  frequent  actions. 

CommandView:  There  is  a  fixed  set  of  time  zones  available.  It  should  be  possible  for  a  user 
to  change  these  (by  using  drop-down  menu,  for  example). 

No  zoom  facility  is  provided  on  the  map  (although  the  user  can  choose  between  a  world 
map  and  a  map  of  Canada).  There  are  some  world  areas  other  than  Canada  where  the  joint 
staff  might  want  more  detail.  The  web-based  interaction  means  that  the  CommandView  user 
will  need  to  scroll  through  multiple  screens  to  obtain  desired  information.  In  many  cases, 
one  wants  to  then  compare  that  information  to  information  previously  viewed.  Then  one 
needs  to  use  the  web  browser’s  back  function  to  find  it,  rather  than  being  able  to  “jump 
back”.  Since  the  list  of  pages  is  finite,  it  would  seem  appropriate  to  attempt  to 
organize/catalogue  the  pages  within  a  hierarchical  directory  structure,  as  commonly  used  in 
many  applications  to  allow  the  user  to  “jump”  to  be  appropriate  page. 

No  search  engine  is  provided,  which  might  help  a  user  determine  what  is  available  on  a 
particular  topic. 

CommandView  does  not  save  the  user’s  workspace.  Therefore,  user  is  obliged  to  remember 
his  last  settings  and  reorganise  his  workspace  accordingly  each  time. 

8.  Aesthetic  and  minimalist  design.  Dialogues  should  not  contain  information  which  is 
irrelevant  or  rarely  needed.  Every  extra  unit  of  information  in  a  dialogue  competes  with  the 
relevant  units  of  information  and  diminishes  their  relative  visibility. 

CommandView:  The  interface  appears  fine  in  this  regard. 

9.  Help  users  recognize,  diagnose  and  recover  from  errors.  Error  messages  should  be 
expressed  in  plain  language  (no  codes),  precisely  indicate  the  problem  and  constructively 
suggest  a  solution. 
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CommandView:  The  interface  appears  fine  in  this  regard. 


10.  Help  and  documentation.  Even  though  it  is  better  if  the  system  can  be  used  without 
documentation,  it  may  be  necessary  to  provide  help  and  documentation.  Any  such 
information  should  be  easy  to  search,  focused  on  the  user’s  task,  list  concrete  steps  to  be 
carried  out,  and  not  be  too  large. 

CommandView:  No  help  is  provided  with  CommandView.  A  help  interface  might  present 
the  organizational  scheme  to  the  user,  and  in  so  doing  provide  a  chance  to  explore  the 
rationale  for  the  site’s  organization. 


6 


DRDC  Toronto  TM  2006-039 


4.  Other  comments 


The  author,  as  evaluator,  had  some  difficulty  accessing  content  in  the  CommandView  Discussion 
Forum.  No  posts  could  be  seen  despite  the  fact  that  according  to  statistics  eight  (8)  posts  were 
available. 

The  tabs  shown  at  the  top  of  the  CommandView  main  screen  were  non-functional  (except  for 
indications  and  warning). 

I  suspect  there  is  potential  for  more  extensive  functionality.  For  example,  one  might  rework  the 
information  shown  on  other  pages  to  standardize  terminology,  presentation  format — a  consistent 
ontology,  using  xml  perhaps. 

The  use  of  a  technology  similar  to  GoogleMaps  might  be  considered  to  allow  quick  changes  in 
viewpoint  for  different  theatres.  The  use  of  accelerators  and  a  zoom  facility  would  be  useful. 

It  is  difficult  to  view  multiple  pages  at  the  same  time  with  the  current  interface.  Use  of  frames  or 
web  browsers  that  provide  a  method  for  switching  between  recently  accessed  web  pages  (e.g.,  the 
tabbed  browsing  provided  by  Firefox  or  Safari)  or  displaying  the  connectivity  between  pages 
using  a  multiscale  interface  (e.g.,  by  showing  miniaturized  versions  of  recently  accessed  pages, 
e.g.,  Pad++)  are  suggested. 

A  second  possibility  (more  long-term)  is  to  incoiporate  interactive  applications  into  the 
CommandView  pages,  make  it  more  of  a  true  portal  which  would  allow  dynamic  control  of 
content.  Thus,  for  example,  relevant  databases  could  be  viewed  from  within  CommandView,  and 
relevant  data  linked  to  current  situation  report  data,  all  formatted  and  displayed  in  a  consistent 
CommandView  format. 
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5.  Conclusion  and  recommendations 


A  preliminary  heuristic  interface  evaluation  of  Comm  and  View  based  on  based  on  Nielsen’s 
heuristics.  In  general  CommandView  displays  key  information  for  the  Joint  staff  in  a 
comprehensive  integrated  format.  However,  key  problems  involve  consistency  of  organization.  A 
solution  may  be  to  provide  a  consistent  interface  by  providing  standardized  terminology  across 
sites  and  dynamic  control  of  content.  A  more  comprehensive  detailed  evaluation  study  involving 
usability  testing  or  other  more  formal  methods  is  recommended  to  generate  more  representative 
and  reliable  results  and  generate  further  guidance  for  improvements. 
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